Skip to content

Persist logs and warnings | Updated sources and analytes | Non-detects and null values | Site limits | Unaccounted datums | Silica | Single timeseries file flag#6

Merged
jirhiker merged 59 commits into
dev/jirfrom
dev/jab
Nov 14, 2024
Merged

Persist logs and warnings | Updated sources and analytes | Non-detects and null values | Site limits | Unaccounted datums | Silica | Single timeseries file flag#6
jirhiker merged 59 commits into
dev/jirfrom
dev/jab

Conversation

@jacob-a-brown

Copy link
Copy Markdown
Contributor

General

  • Persist logs and warnings or users
  • Convert mg/L as CaCO3 to mg/L for HCO3- (conversion factor = 1.22)
  • Report skipped items for summary (mostly Null values as they can't be summed or compared)
  • Set non-detects to 0 for time series
  • Only count sites toward the site limit if it has an observation
  • Remove sites with datums that are not accounted for. This is mainly for WQP when the datum is UKNOWN
  • Added silica as an analyte to query
  • Don't include alkalinity, bicarbonate (or any iteration thereof) for Bicarbonate as that's an alkalinity measurement and not necessarily bicarbonate data. I'll get more information from Talon to see if it should or should not be included.
  • Added --single_timeseries_file flag for user for a single combined timeseries file

DWB

  • More analytes added for DWB
  • catch non-detects. Keep for time series remove for summary

** WQP **

  • Only use sites in NM

** NMBMR **

  • Use updated (/latest) API
  • Only use Groundwater other than spring (well) site types

This update was made because the FROST server no longer accepts
MultiPolygons as valid WKT inputs. This change will allow the
ST connector to convert MultiPolygons to Polygons before sending
them to the FROST server. However, if the MultiPolygon is not
continuous, then the MultiPolygon won't be converted to a Polygon
Sometimes a WKT string is passed, so called poly.geoms will cause
an error. To fix this, a MultiPolygon is converted to a Polygon only
if type(poly) == "MultiPolygon"
site.name was printing the site name, but the id is used everywhere
else. Now, when a site has no clean records or parent records, the id
instead of the name is printed
If the enumeration starts at 0 then limit + 1 sites will be collected
Before this update all sites were counted toward the site limit, even if
they had no associated records. This change updates the site limit
to only count sites that have associated records.
The tests were failing because the combined file would be created
in some situations but not others. Because of this, the combined_flag
needed to be set for each source appropriately
This allows any transformer class to communicate to users in the same
way that source classes do. This is useful for debugging and for
communicating crucial information to the user.
This update indicates that a list of SiteRecords is returned, rather
than just a list (what was previously documented)
All parent_records are site records. To clarify this, parent_record has been renamed to site_record.
This information is evident when the files are named for the location,
but needs to be included in the single timeseries file. For ease of
development, and for being explicit, this information is now contained
in all timeseries files.
Some locations may have more than one record, making dump_single_timeseries write the headers to multiple rows. This change ensures that headers are only written to the first row.
These methods are not yet implemented, so warnings/messages are printed
every time. These have been commented out for now, but should be
implemented in the future.
… file

This update allows the user to specify if they want separate timeseries files for each
timeseries, or a single timeseries file for all timeseries. This update
also includes documentation in the README
Non-numeric results cannot be compared or summed for the purpose of
summarization. They are kept for timeseries data
jacob-a-brown and others added 29 commits November 4, 2024 17:45
Some units are reported as mg/L CaCO3**
…n omitted

This is because the unit conversion was being done on items, which are used
for summaries, whereas the conversions needed to be done on the records themselves
DWB is still in progress. This commit is to push to GitHub to
preserve the work done so far.
@jirhiker jirhiker merged commit a5f8d15 into dev/jir Nov 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants